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Program Analysis and Monitoring (PAM) in Writing is a 
decision-support tool which provides the principal with an 
informative report, the Program Analysis Report, .that 4 identifies 
needs and -suggests solutions to problems in educational writing ^ 
/programs.* The Report relates information on student achievement in 
/ writing to information on how resources are distributed and used for 
,>cl&sterbom instruction. It provides the principal with an agenda for 




of 1981&to several thousand Putnam, Suffolk, and Westchester County 4 
third and sixth grade students. Two reviews resulted in the selection 
of four responses per grade level for each' assignment to be used as 
model answprsl in ratiftg^ This paper inc^udes N illustrations of the PAM 
in Writing Monitor Report*- and jictual -datfc which appears in the PAM 
Program Analysis Report. (PN) 
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Program Analysis and .Monitoring (PAM) in Writing: Practicay Implementation 
'Program Analysis and Monitoring— Rationale / - 

Whether , or not it is'now a cliche, to say that the/public haspbecome 
unprecedentedTy critical" and demanding of our schools wjiile simultaneously 
tightening the educational purse strings, it is nonetheless incurrafent upon 
those in the business of educating children to .improve, edUcatiojCwhenever 
possible- arid to see that schools perform to their maxtrnurrv capabilities, \ „ 

To assure such improvement and performance reoujres a clear percep- 
tion of the "educational (i.e., ^instructional) procWs, . an understanding 
"pf" the locus of responsibility for the implementation bf instructional pro- 
grams, r and tha^riWitfgpess and ability to assume arid -apt on^hat re$porisi-<, 

\ % \ vr , ^I.f .therfe- are shortcoming? in substantive instructional practices 
***(fi'9Cr purveyance of inaccuracy, • confusion*. of procedures? subversion of ' , 
• skills)/, 'teachers may thenfselves need remedial <if(str.uct ion. The diagnosis 
'•v, , of stifti shortcomings, would entail o.bserv.at ton* within t*he classroom. But' 
-\ , , ; such mis instruct ion WouTd of 'course be rare. tyhfre suc^ problems are min- 
"4 ima.l, students' /afcfuisitioir of th^ ba'sic'skil JsririyoJvfed in'reading, "writ- 
ing-, and. arithmetic' (• and by extension the bas^j^xonceptjs and skills in 
such subjects as. social studies and' science);!^ a -matter of exposure and 
'practice. ^ This involves time. ^ To the exteht 5hat sufficient timers de- t 
voted* to exposure and* practice in-the use of a 5 skill 0£ concept, acquis^, 
, tion°will occur. Rates of acquisition vary 5 .[out all learners require sorrte- 



1 , * 



time for acquisition, The most effective s'chool is the ORe wfyicb optim- 
izes acquisition time^for the greatest proportion of :Tts students. 



It behooves schools, then, to pursue Vithin-cl ^isroem practices which 
improve the way- time is sp^nt. Such practices of course exclude consid- 
eration of factors external to. the -school' or beyond tlie school's control*^ 
The school, after all, s.eryefs the communnt^l and is in* some sense aoextert- 
' sion of that community. It i«s .therefore vain. to expect the school to */ 

transform the* CQmriiuni ty. But it is not vain t9 expect 'self -transformation 
. by the schcfol.. To do so, however, requires useful and timely information 

r* n $ tf^ capacity to use that information:^ * \^ '!j 

Th'at^capacity entails someone in :a position df responsifcil ity*.^br 



. Vhe schools instructional programs. In a. typical school that pers<jfi is 
, *nyt a teacher; it is usually the principal*, a department chairman, tit a 



subject area supervisor or coordinator. 



only rarely a teacher is clear: The teacher's perspective—not to' men- 



is his own classroom. No mat- 
ssr.oom practices, nothing Jit n the 



tion physical* and tefrooral restrictions 
ter how excellerit a gliven teacher's eld 

r\aUire of classroom teaching assures ttfe permeBt^on of ' a teacherfs in 
fluence .beyond his owq classroom. TheHe'is no assurance that th& students 
of art excellent Jteacher. would encounter" 
rooms or will ^encounter it in subsequen 

teacher possesse'd of a reformer' s zeald there is scant opportunity for any 
teacher to evaluate.another teacher's performance. Further, e\fep given 
.such evaluative opportunity, one teacheij is not* usual ly *in a position to 
ask or require another teacher^s reform] 



The reasoh why such a per/on is 



Someone, then, usually the" principal and almost *never a teacher, must 
be in the position of program manager. Someone must have access to or the 
capacity to acquire information about the -program plus the 'authority to 
act, to make decisionsjn response to such information and to require the t 
implementation of those decisions.- ' It must be emphasized that mere occu- * 
pation of'such a position is not equivalent to management- of a school's 
instructional program: Management involves the willingness to carry out 
the functions implicit in the position. This raises two isstfes. The first 
is the issue of willingness and the second' is,thejssue of capabil ity. 
To some extent will ingness to* be a program manager, 'tp do the job, is cf 
matter of personal temperament,. charaqter, "experience, and motivation. 
Where the first three of these militate against or .discourage action, 
where, in other -words, there iX no intrinsic motivation in the person,- r r 
external- motivation will be required, ;fos instance in the* form af,eneour-\_ 
agement frdm the school- board or central administration. But motivation* 
may-exist uncoupled .with any awareness of -.how \b manage educational pro-' 
grams.. ^Here \s where willingness alone vs insufficient. 'HelpfuT'tools 
and/br training are^lso required. .One such tool is "PAH— Program AnaTysis 
and Monitoring-- in Writing. . * / 

,v ^ 'PAM in Writing and Decision-Support 

' PAM' in Writing is a decision-support tool which. provides the prin-' 
cipal with an inYormative report, the Program Ancrtysis Report, that helps 
identify needs and suggests solutions to problems in writing programs. 
The information provided by this report, enables the principal* involved 
in .program' management to make ^programmatic decisions.- For the principal 
less involved in program management b'ut interested^- in acquiring and devel- 
oping^management skills, the RAM Program Analysis Report is useful in 
building awareness of how to think about the instructional program. 

The PAM Program 'Analysis Report Js 'timely as well as useful. Since 
the report is available in the fall or winter, and then again in the 
spring of the year,- it can be used^for both^ formative; and summative eval- 
uation. For example, if.the fall report reveals potential problems in 
the program, steps may be taken immediately'to specify the problem a,s ex- 
actly &s possible and then to attempt to correct it with appropriate ^ 
program modifications. Receipt of the spring report is an opportunity to. 
determine the efficacy of earlier corrective measures as well ,as to ev^l- • 
uate the year's efforts. - - . ¥ p - * . , * ^ 

The utility of the Program Analysis Report lies in the nature and 
juxtaposition of its information. The report" relates information on stu- 
dent achievement in writing to information on how resources arg distrib-^ 
uted and. used for classroom instruction. 
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PAM PROGRAM ANALYSIS REPORT— BASIC STRUCTURE. 

For PAM ( 'in Writing," the 'achievement information ir* the Program Anal- 
ysis Report is of -two "types: average JioMstic scores and diagnostic indi- 

' cators of classroom needs. Bat hr indicators are b^sed^on the 'PAM Writing ' . * 
Test. This test requires" students to*r\espond to one Writing assignment. 
Student responses are hoi ist-ical ly scorld by two raters who compare the re- 
sponses to model. (100%) answers, there ''are five alternative PAM writing : . t 
assignments with,- typically, four model answers for eacfy at grades 3 and 6. 
Thus, students in grades 3 and 6 respond to identical assignments, but their 
responses are rated against model answers -for .their' own grade levels. 'The 

(average of the two raters' assessments' is the student's score! *t 

Diagnostic indicators are also based on the J%P Writirfg 'Test. Indi- 
cators are expressed as the percent of -stude/its in^a qlass with instructional 
"needs, in any of five categories of writing-related^ skft^s: Rhetorical Task, 
"Relation of Parts to a Unified Whole, Worcf'Choice, Synt%, and Mechanics. 
Classroom diagnostic indicators are derived from the classroom teacher's 
assessment of ' individual student weaknesses or neecTs. \m 



Schools also have the*pptions of t receiving student performance in-> 
formation based on Statewide writing measures. These Jnclude The Prelim-- 
inary Competency Test {for gr'ades 8 and 9) and an elementary-level test 
which is presently under develppment. Additional diagnostic indicators 
are also-available at .the. school* s discretion for more detailed precise* 
analyses of writing needs. 

Regarding "information on the distribution and use of programmatic 
resources, the -RAM Program Analysis' Report is quite comprehensive. It % 
reports use and relative importance (by classroom) of a number of resour- 
ces for student placement, assessment, "and evaluation; frequency and var- 
iety of -writing-related activities; and time allocations by 'instrctctionaT 
modes >or configurations. * 

o 

v • • 

Information Veported on these programmatic variables is collected 
from' "teacKers by means of a survey instrument, the PAM Teacher .Question- 
naire. Though comprehensive,, the questionnaire may be completed, in less 
than one ho A ur. . - ... 



* x Th£ achievement and program resource information juxtaposed in the PAM Pro- 
gram Analysis -Report : faci 1 itate decision-making at the program leveT. If 
student achievement is not up jto expectations/ decisions must- be 'macte' about 
* appropriate action. If goals haye been set regarding the use or distribution 
•of program resources, the PAM Program Analysis Report'will show compliancy 
with or departure ifrom such goals. . Again, decisions must follow in response 
to this information. Varying combinations of satisfactory and unsatisfactory 
results on both achievement and resource use and distribution will of course 
give rise to.rpany kinds of decisions. If, for example, achievement is sat-; 
isfactory but program resources are apparently being squandered, the program ' . 
manager is faced with decisions. .Decisions are also required if resources 
appear torbe used efficiently 'but achievement Js disappointing. Sometimes 
the information* in the Program Analysis Report will function as the beginning 
of a process^ Df diagnosing.program needs or* problems which will in time re- 
sult- in decisions to modify the program. The point is this: Rational decision- 
making requires, an information baseband the PAM Program Analysis Report provides 
such a base very systematically and comprehensively, - 

TKe^PAMJProgram Analysis Report, then, provides tfhe principal with an 
agenda for assessing program needs" and determinihg approprtate'respgnses. 
Regarding a school's writing -program, responses to needs might involve modi- 
fication lrfthe use os- distribution of the resources analy2edin the PAM 
report^ For # pxample, a'decision might.be made to increase or decrease the 
amount of tirffe allocated for*writing instruction. Perhaps students might 
be grouped differently for writing^ instruction. Or perhaps the types of 
writing assignments might be varied. Such decisions could be'made on the 
bases of the PAM report and discussion with . instructional staff. .Other 
kinds of decisions would require more intensive investigation and, ultimately, 
greater intervention in writiirg classrooms.' Such investigation might deter- 
mine the existence- of instructional staffing needs in the areas of general , « 
instructional techniques and instructional' planning, classroom management 
procedures, or content-specific techniques.or information. Again, such needs 
could only be discovered through rather intensive investigation or diagnostic 
effort by the program manager. Responses are available to all such needs, 
but needs identification must^precede correction. 

With the PAM Program Analysis Report,, the principal can take, the lead 
in improving his school's writing program. 

PAM in Writing Development and Implementation w 



PAM in Writing is an extension of Fwl in Reading. In other words, 
PAM ir\ Waiting is an outgrowth, both conceptually and technologically, 
of Program Analysis ajrid Monit6ring in Reading. The pfincipal features of ° 
PAM in Reading are a' Program Analysis Report relating achievement and in- 
structional variables, a classroom-level achievement report (the Moretor 
Report), Literal Comprehension Achievement Monitors .(a battery of 24 mul-* 
tiple-choice, paraphrase-based reading tests Tev^l led by reading passage 
difficulty), and a Teacher Quest i oniia-i r& \or gathering process informa- 
tion. A major strength of PAM in 'Reading is its scale of, reading achieve- 
ment, permitting measurement * on the same metric across grades^ The anal- 
ogues of theso .components in PAM in Writing are the Program Analysis Report,, 



the Monitor Report" on student writing achievement at the level of the 
classroom, -the 'PAM Writing Test, and a Teacher Questionnaire. 

PAM in Reding was developed withfji the New York State Education 
Department from 1977 to 1979; from 1979 to early 1981 PAM in Reading was 
extended and improved (as Title I PAM) under a contract with the U.S.'* 
Department of Education. PAM in Writing has been under development since 
1980 as a cooperative effort involving State Education Department <suopor-t 

of the Putnam-Northern Westchester BOCES. T& date the PAM 
Writing Test, the Teacher Questionnaire, model answers, directions for 
raters, and the computerized Monitor Report are complete. The computer 
software for the Program Analysis Report is under development. 

The major problem in developing a tool which wpuld yoke writing 
achievement information .with writing programjDractices "was the creation, 
of an* acceptable measure of writing achievement. The measure would have 
to.be valid, which ruled out any objective-type or non-pfcrformange meas- 
ure. The students to be rated would have to produce writing samples. Avail- 
able rating options included teacher-assigned grades, primary trait stor- 
ming,' holistic scoring without model answers, and holistic scoring wtth 
model answers. The last of these options w.as chosen because of its ad- 
vantage over all the' 'others. Teacher-assigned grades were ruled out as 
too idiosyncratic. Primary trait scoring, as illustrated by t\\e work of 
the National Assessment of Educational Progress, was rejected because, of 
its ^reat complexity and associated high, cost. Standard' ho/I istic scoring, 
where raters assign papers to one of several categories, for Example on ■ 
a 4-3-2-1 basis, was rejected, even when*models were employed, because of 
the relative grossness of the categorization or attempted scaling and the 
difficulty of generalizing or averaging from individual performance to 
group performance. 

"Holistic scoring based Qn $ scale of zero to 100 and using 100% 
iftodel answers was selected for two main reasons: (1) The scoring which 
.would result would'have very attractive properties for program analytic 
purposes, and (2) the State Education Department, as part of its Regents 
Competency Testing program, had succeeded in refining this methodology 
to a very high degree. In a word, it worked very well. Additionally, 
detailed, though not overwhelmingly lengthy, guidelines were available 
as a basis for the slightly modified rating directions which wereeven- 
tually developed for PAM in Writing. 

development of the five wrifing assignments was guided mainly by 
the desire to assure that students' would, respond; would be able to pro- 
duce a writing sample. Accordingly, one concern was to avoid the use of 
topics which would be unfamiliar to students (and many potentially in- 
teresting and st inflating* topics run the risk of being foreign to some 
students) and would, hence, stifle their output. In other words, the 
PAM 'measure of writing achievement was not to be a measure of invention 
of imaginative fertility. Further, .topics were to be developed on which 
students' in different grades could have an equal chance to respond. Con- 
sequently, the five PAM wri£ipg assignments feature* the mundane and fam- 
iliar, (e.g., describe your favorite game, describe your favorite person). 
In addition to. the familiar topics, the five assignments are quite specific 
in' their directions. Again, the aim was t!o assure a response if possible. 
The -following illustrates the typical structure of a writing assignment \ 
from the, PAM Writing Test: . - ( 




"Tell me about your favorite animal. Write three 
~ paragraphs. Be s.ure to tell me— what it looks like, how 

big it is, whejce it lives, what it likes to eat, 'and . , 

e why it is your favorite. 11 / 

Once the assignments were constructed, .they were administered, in 
Way and June of 1981, to several thousand grade 3 'and ,6 stadents represent- 
ing school districts? i.n Putnam, Suffolk, and Westchester counties. , All * 
respopses were reviewed\and the best ten or twenty responses from each 
grade and for each assignment v/£re selected for a second review. This ^ • 

review resulted in the selection of an average of four responses per \ * , 

grade level for each assignment. The responses selected in this final 
review underwent minimal editing (nearly all related to mechanics): they 
are new the model answers used in rating. In otHer words, the model 
answers are high-quality student responses. ~ * 

The PAM in Writing Teacher Questionnaire had the advantage of a 
successful model in the PAM in Reading Teacher Questionnaire. That 
questionnaire had established certain basic categories of variables which 
were both quantifiable (i.e,, measurable) and controllable by the school-. 
Such major categories as allocated instructional time;'- resources and 
materials for placement, .assessment, and evaluation; and instructional * 
grouping had only to be f-lushed out with specifics pertinent to writing 
'instruction. To these categories of items were added a-thorough- list of 
various' types of writing assignments. The questionnaire was 'reviewed by 
practitioners in writing instruction and management, and the final ques- 
tionnaire was precoded £o. facilitate 'the processing of information., The 
major kinds of program variables covered by the* Teacher Questionnaire 
are listed below: * » • 

-J - 

Use of instructional resources Ttffie allocated for, writing instruction 

Methods of grading (ranked)*, " * Time allocated by type of writing instruction: 

Methods of assessment (ranked) ■ One-to-one 

Information used for pupil placement , Supplemental, extra-class, specialist^ 
Frequency and variety of writing 
assignments focusing on— 

Audience t 

Purpose 

Form 

Frequency of writing assignments 

by.length of assignment 
Frequency and variety of prewriting 

activities 
Frequency an& variety of revising 
• -activities 
Number and type of evaluation 

strategies 
Class size • • 



Supplemental, extra-class, aide 
Time lost for supplemental' writing 'instructibn / 
Time spentMn Whole Class, Spontaneous Grolip, 
t -and Stable Small Group Writing Instruction 
•in' the following categories: V, 
"Direct Instruction from C/lassroom Teacher 
Other Adult Working with Pupils 
Pupils Working Independent ly-^-.Teacher: 

Available' ' ■ ;> 

Pupils Working. Independently—Teacher 

Unavailable 
.Pupils Working Toqether--Teacher Available 

Pupils Working Tt)gether--Teacher Unavailable 



' Piloting PAM in Writing 



In late fall qf 1981 the PAM Writing Test was administered to'ap- - 
proximately 1,000 students in 40*classes at grades 3^6, 7,. and 8. Sub- t 
sequent.ly, teachers were brought together in groups under the direction 
of a consultant experienced in writing assessment >and holistic scoring.- 
The teachers 'were trained in the application of the rating techniques for 
tte PAM Writing Test. Then each class set of student responses w&s rated 
m by, two teachers; *in eve 1 ^ case the. second rater was the classroom teacher, 
Ratings were then averaged. Where time permitted, classroom teachers 
reviewed student responses for diagnostic purposes. At these sessions' 
♦teachers also completed the TeacheV Questionnaire. 

W / * 

Prior to testing, some concern had been expressed, by teachers and 
other educational professionals; that students in *grade 3 either should 
not, be tested at all of should or^ly N be tested toward the end. of -the school 
year.. The first of these concerns seeded to assume either that some harm 
would cofne to grade 3 students, through tyje testing or that; perhaps test- 
ing grade 3 students in Writing would be presumptuous in the light of com- 
mon practice in writing instruction at>t that grade level. However, thje 
test administration experience did not bear out this concern. No evidence 
of student ^problems, ^ psychological, or physical, was reported. The sec- 
ond concernV that studehts should* be tested onlyat the end of tf\e thir,d 
grade, had Been voiced by many third-grade teachers, 1 typical ly' on Ithe 
grounds thatVtudents had not yet received instruction in some of the 
technical aspects of writing. This -concern revealed a lack, of understand- 
ing of the purpose of PAM in Writing. If the teachers were correct, then 
knowledge of certain^ref inements- of writing'tf'ould indeed not be manifested 
by third-Naders early in the'year; their responses would be rated rela- 
tively low, accordingly. But readmini strat ion of the PAM Writing Test at 
the end.of the school year, after the relevant -instruction had occurred, 
should then reflect £he 'benefits of that instruction in terms of increased 
achievement. In fact, however, lack of instruction notwithstanding, the^ 
thir&-grade students were in no wise stymied by the writing assignments.* 
They were able to produce. It is expected, of course, -that a spring test- 
ing .will reflect growth, no doubt resulting from instruction. 

Handouts used in the presentation of PAM in Writing at the annual 
meeting of the American. Educational Research Association include illustra- 
tions of the PAM in Writing Monitor Report and actual data which will ap- 
pear' in the PAM Program Analysis Hreport. These handouts are available 
from the Bureau of ESC Educatiot^Pl anning, 481 EBA, Albany, New York* 12234, 
In addition, copies of the following materials are also available from the 
same address: , fc ' 



Monitor Report (mock-up) fc 

Program Analysis Report (mock-up) 

PAM Writing Test (sample) ■ 

Teacher Guide for the PAM Writing Test (sample) 

Model Answers (booklet) 

Rating and Score-Recording Procedure 

Teacher Questionnaire 
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CLASS^' 



BCD - /Avg> - 




Average holistic score . 60.0 ' 51.0 - , 66.1 74.3 63. 

Pupils * 23 '** . ' 19 ' 22 > n "2^.5 

W % • 

Minutes per week allocated * 140" ^ , 90 300 80 15?. 5 ^ 

for writing instruction - % | " . ■ / 



Average minutes per week 4 
. rercgived by pupils — 

Whole Class Instruction 
J)irect Instruction ffpm^ 
teacher 



90 . - * 9tt 150 80 



Other Adult Working with < : % 

> ' .Pupils \ - r-. 45 % - 7S . - 150 

* Pupils Working Independently v . ' , . 

--Teacher Available . , 105 110 _ 150 .80 



Pupil Working Together— 
Teacher' Available ' ' \. 105 75 • _ 150 



: v Number- of in-clas,s "Writing 
Assignments per- Month- «* 



21 ' • ■ "25 . 21' 5 • " 18 
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Use of instructional resources' \ Time 
* t , Methods of grading (ranked\ ; Time 
'Methods of assessment pranked) 
Information used for pupil' placement 
Frequency and variety of writing 
assignments focusing on--, " Time 

Audience Time 
Purpose 
Fotti 

Frequency of writiifg assignments 0 . 

by length of assignment 
Frequency 'and variety of prewriting 

activities * 
Frequency and variety of revising % 

activities * 
Number and type of evaluation 

strategies . ; 3* • 

Class size 



allocated fo& wfciting instruction • 
allocated by type of writing instruction: 
One-to-one, 

Supplemental, extra-class, specialist 
Supplemental, extra-cl^ass , aide 
lost for supplemental writing instruction ' 
spent in Whole Class, Spontaneous Group ,^ 
and Stable Small Group Writing Instruction 
in*the following categories: 

Direct Instruction from Classroom- Teacher 
Other* Adult Workiiig with Pupils 
Pupils Working Independently — Teapher 
Available 

Pupils Working Independently — Teacher 

Unavailable . . 
Pupils" Working Together — Teacher Available- 
Pupils Working Together — Teacher Unavailable 
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